Thanks for getting this started, @mkv123! Some initial thoughts/questions:

• Once we resolve the question of exactly what RegExp synatx to support, we'll need to validate input against that subset.
• Should the regexp itself be allowed to be an expression, or should we require it to be a literal value? I lean towards the latter, since that would allow us to precompile the regexp at parse time.
• While the more general match-group-returning form could technically be used for things like boolean tests or search/replace, I think it would be nice to have specific expressions tailored for these very common, simple use cases.

cc @mapbox/gl

Hi Anand and thanks for the input.

1. "...exactly what RegExp synatx to support..." : Are there thoughts on what the best way to do this validation would be without implementing a full regex parser in javascript & native? Would documentation instead of enforcement be an possible approach? There are already documented differences between native and js.

2. "should the regexp be an expression": While I don't see a tremendous need right now for the flexibility of allowing the regexp to be an expression, as a user I would find it a surprising limitation. Also many language implementations, including javascript engines, cache the compiled expressions internally, yielding practically no performance improvement from having the expression precompiled vs dynamic. Should regex compile time become a performance bottle neck, similar caching could be implemented in native. Imho, there should be a measurable benefit for adding an extra limitation, and I'm not entirely sure that would be the case here.

3. "it would be nice to have specific expressions": How about "regex-search", "regex-match" and "regex-replace" to return boolean, array of matches and replaced strings respectively?

Also many language implementations, including javascript engines, cache the compiled expressions internally, yielding practically no performance improvement from having the expression precompiled vs dynamic.

This is the case for ICU, which underlies the most obvious regular expression implementations that mbgl or the native SDKs would use.

Updated the pull request to use "regex-match", "regex-test" and "regex-replace" for the three different uses. Added basic tests for them as well.

As regex replace is a string method in javascript and in ICU, I'm not entirely sure what would be the best order for the "regex-replace" arguments. I do think the regular expression string should be the first argument on a regex operation. Current order in implementation is expression, flags (optional), replacement string, original string, which happens to match the order for python's re.sub() for example.

I've looked at the regex standards and hunted for a nice comparison between them to establish what the differences in supported syntax are, but haven't really been able to find one.

For reference:

• ICU: http://userguide.icu-project.org/strings/regexp
• ECMA: https://www.ecma-international.org/ecma-262/6.0/#sec-regexp-regular-expression-objects

I've looked at the regex standards and hunted for a nice comparison between them to establish what the differences in supported syntax are, but haven't really been able to find one.

This table on Wikipedia is a good starting point. For the purposes of map styling, I think the most notable differences are the lack of (fixed-width) lookbehind, named capture groups, or Unicode character classes in JavaScript regular expressions. For example, \w matches only [A-Za-z0-9_] in JavaScript but all analogous Unicode codepoints in ICU ([\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\u200c\u200d]).

JavaScript also has a hack in which [A-Z] matches the Turkish İ.

/ref #4089 (comment)

I think detecting and preventing the use of syntactic features that aren't supported by both ICU and JS is probably okay, but it's less clear to me what we can/should do about the differing behavior for things like \w that @1ec5 mentions in #6228 (comment).

In terms of API design, how about:

• ["regexp", string, flag1, flag2, ...] (the flags being optional), produces a value of a new type regexp. E.g., ["regexp", "[a-z0-9]+", "i"]
• Search with ["match", haystack: string, needle: string]: boolean or ["match", haystack: string, needle: regexp]: false | array<string> (so that either could be used as a boolean)
• Replace with ["replace", src: string, query: string | regexp, replacement: string], with the replacement expression evaluated with a ["group", number] expression available. E.g.:

["replace",
    "philadelphia, pa",
    ["regexp", "(\w+),\s*(\w+)"],
    [ "concat", ["group", "0"], " ", ["upcase", ["group", "1"]] ]
]

match is already used, and in general, I think we ought to start namespacing related groups of functionality. How about a family of operators, "regexp-match", "regexp-search", "regexp-replace", etc.?

Of course, it's not clear exactly what each of those should do. There's a number of possible use cases, and different languages make differing choices about which are conveniently built in, or which names to use. I surveyed JS, C++ <regex>, and Racket (as a lispy representative):

Other random notes:

Note that a full-fidelity match result is array<string | null>, not simply array<string>. This reflects the fact that capture groups may not participate in the match at all; in JS, the result of /a|(b)/.exec("a") is ['a', undefined].

For replace, I'm not fond of "group", as it would require introducing regexp-specific state to the evaluation stack and we'd need to validate that no "group" expressions appear outside a "replace" context. I'd prefer to:

• Introduce first-class functions, so the replacement can be represented functionally
• Use the standard $n meta-syntax. (This is functionally the same as "group" but the implementation would be more localized.)

The first is preferred, since it's a generally useful feature, and we try to avoid meta-syntaxes.

For completeness, here’s a corresponding table for the various regular expression–related APIs in Objective-C and Swift:

I tried regex-test in a filter and it works very well, great work! Hope this will be added to a standard release asap.

One thing I'm missing is the possibility to make the regex case insensitive with the 'i' flag in RegExp()

• Introduce first-class functions, so the replacement can be represented functionally
• Use the standard $n meta-syntax. (This is functionally the same as "group" but the implementation would be more localized.)

The first is preferred, since it's a generally useful feature, and we try to avoid meta-syntaxes.

I think we should hold off on first-class functions until we've seen one or two more concrete use cases where we'd want to use them. (I can hypothesize such use cases, but first class functions add enough complexity that I'd want to have some confidence that they're going to happen in practice.)

Agreed on ideally avoiding a meta-syntax (though at least this one would be a pretty well-established one).

I'm not fond of "group", as it would require introducing regexp-specific state to the evaluation stack and we'd need to validate that no "group" expressions appear outside a "replace" context.

A variation that would address the former, but not the latter, issue would be to have ["regexp-replace", pattern, haystack, needle] bind $0, $1, etc. as let variables in needle's evaluation stack.

I have tried regex-test a bit more as a filter, and it works well when used as the only filter, e.g.:

filter: [ "all", ["regex-test", "VN-*", ["get", "property1"] ]

but when adding more than 1 filter, e.g:

filter: [ "all", ["regex-test", "value*", ["get", "property1"]], ["==", "property2", "value2"] ]

it breaks with the following error:

evented.js:121 Error: layers.id-b233f807-e131-4b3a-a15e-6d03a5346d40.filter[1][0]: expected one of [==, !=, >, >=, <, <=, in, !in, all, any, none, has, !has], "regex-test" found
at Function.module.exports.emitErrors (validate_style.js:12)
at t._validate (style.js:888)
at t.addLayer (style.js:534)
at e.addLayer (map.js:1171)
at eval (MapBase.vue?2507:92)
at new Promise ()
at new F (_export.js?90cd:35)
at VueComponent.addLayerAsync (MapBase.vue?2507:91)
at Object.boundFn [as addLayerAsync] (vue.esm.js?efeb:189)
at eval (LayerSettings.vue?4a1a:499)

SOLUTION:

Added

"regex-test": { "doc": "" }

in filter_operators in src/style-spec/reference/v8.json

Any recent movement on this? It would be a very valuable addition.

I'm going to close this PR because it's stale. Let's continue the discussion in the original ticket #4089.

Too bad, it's a perfectly working solution (with my above addition) which I have to manually merge in to a forked version every time a new version of Mapbox is released.

@anderslarssonvbg please read this ticket, and #4089 (comment) for rationale.